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Purposes 


The purpose of this paper is to open the methods, construction and assumptions of a new 
performance benchmark to examination, critique, and improvement by technical experts. 


The paper begins with known technical questions about this new benchmark. In includes 
suggestions made by technical advisors. It then details the construction of the 
benchmark’s measures, and reports the relevant statistics. 


KSDE’s new performance measure, the effectiveness rate, measures school districts’ 
effectiveness in enrolling their students in postsecondary education for two continuous 
years after graduation, or in getting students certified in an industry-recognized skill. 


Its companion benchmark, the predicted effectiveness rate and the focus of this paper, 
uses linear regression to describe districts’ mean effectiveness rates after adjusting for 
risk factors largely beyond the districts’ control. By comparing districts’ actual 
effectiveness rates to their predicted rates, while accounting for student and 
environmental risk factors like cumulative poverty, the model assumes that the residuals 
contain some comparable measure of district performance. 


With some important qualifications, KSDE’s calculation of an effectiveness benchmark 
is a simplified application of a value-added model. In the late 1980s through the first 
decade of the 2000s, most notably with papers by Willms, Raudenbush, Bryk, Sanders, 
and McCaffrey,’ scholars developed various accountability models that controlled for an 
assortment of student and environmental background characteristics. 


The current benchmark model differs from these earlier value-added models in notable 
ways: 


1. It does not use state assessments as its dependent variable; 
It does not attempt to isolate teacher effects; 

3. It does not threaten district or school personnel with punitive measures, but seeks 
to motivate them with demonstrably achievable measures; 

4. Many of the variables it uses as controls are based on proportionate measures of 
student exposures to particular statuses, for example, how long students were 
eligible for free lunch, or how long they were classified as having a disability; and 

5. It uses multiple cohorts (five of them), in its dependent variable.” 


It also shares some problems with the previous generation of value-added models. Most 
notably, it has no direct or observed measures of district practices. It just assumes they 


1 For a brief summary of these models and the scholarly debate about them, see Raudenbush, S.W. 
(2004), What are value-added models estimating and what does this imply for statistical practice? Journal 
of Educational and Behavioral Statistics, spring, vol. 29, 1, 121-129. The same journal volume also has 
articles from other scholars debating the features of value added models. 

* Raudenbush, op. cit., recommends multiple cohorts. 


are included in the residuals. It also assumes that these practices are not correlated with 
other school and student-level independent variables. 


Technical Questions 


So far, KSDE staff identified the technical questions below. We are including the 
responses and additions made by the first group to review the model,’ the assessment 
program’s Technical Advisory Committee (TAC). 


1. 


Is the current application of a simple, ordinary least squares regression model 
appropriate at the district level? 


The TAC had no objections to the linear regression model applied at the district 
level. 


KSDE has plans to develop effectiveness benchmarks at the school level. Applied 
at the school level, is the appropriate model a hierarchical model, specifically, a 
generalized linear mixed model that would treat the district level variables as 
random effects and the independent variables as fixed effects? 


The TAC recognized that the school-level model should be hierarchical, with a 
provision for capturing and setting aside district-level effects. It did not have an 
opinion about whether the model should include a district-by-school interaction 
term, or whether the district level effects should be treated as random effects, or 
the independent predictors as fixed effects. The TAC did suggest getting advice 
from the Regional Education Laboratory technical advisors already working with 
KSDE on this project. 


At the district level, there was very little missing data, but there may be more 
cases of missing data at the school level. What is the best way to handle these 
cases of missing data? 


The TAC recognized that missing data at the school level is more likely to be a 
problem than at the district level. While inserting mean values might be an 
acceptable solution, the TAC noted that there are more sophisticated tools for 
imputation of missing data and that KSDE should consider them. 


KSDE’s answer: 


SPSS offers a specialized tool for applying algorithms to fill missing data. It costs 
$847 per user with a yearly renewal fee. Is the potential improvement in accuracy 
worth this investment? 


3 The Kansas Technical Advisory Committee (TAC) works with KSDE’s Career, Standards, and Assessment 
Services to guide the State’s K-12 assessment program. They reviewed the model at their October 18", 
2017 meeting. 


Based on research measuring the developmental effects of poverty, the model 
uses longitudinal measures, proportions of student time in particular statuses, 
rather than cross-sectional snapshots of student statuses, as independent variables. 
Do we need to demonstrate that the longitudinal measures are more accurate or 
appropriate? Do you see or anticipate any problems with our current longitudinal 
construction of these independent variables? 


The TAC felt that the longitudinal use of student measures was generally more 
reliable than variables constructed from cross-sectional aggregations. 


What problems should we anticipate from our use of this model at the district or 
school levels? 


The TAC raised these additional questions: 


I. Howcan you be sure that your effectiveness measure is sensitive enough 
to describe true differences in performance between high schools? Would 
another outcome measure, for example, the Assessment Performance 
Index, identify the same set of schools or a different set of schools? How 
would you explain the differences, if you found them? 


KSDE answer: 


Because the dependent variable measures postsecondary effectiveness 
rates, not academic performance, we expect that two models with the same 
independent variables, but with differing dependent variables, will 
produce different performance results, but with a lot of correlation, too. 


The distinctive thing about this model, no matter what the dependent 
variable is, is that it controls for factors beyond the schools’ control, like 
cumulative poverty. Previous school performance measures, like Adequate 
Yearly Progress under the No Child Left Behind Act, did not account for 
these influences. They confounded the influences of poverty with measures 
of academic performance. Ideally, over time, we expect that the Board’s 
initiatives, like the Individual Plans of Instruction, better vocational 
counseling and postsecondary referrals, better early childhood 
environments, and new school models, will result in higher effectiveness 
rates and measures that differentiate performance between high schools 


with actual measures of what schools are doing, not just inferred 
differences. 


2. By leaving the non-significant variables in the model, is statistical noise 
obscuring some of the signal from the significant variables ? 


KSDE answer: 


Originally, a workgroup consisting of representatives from the districts 
and KSDE staff identified the independent predictors. They expected that 
the initial nine variables they identified contributed to lower school 
academic performance. By using these variables in the first models 
predicting effectiveness rates, we are testing our assumptions about causal 
factors. Which ones matter most? Which least? It is important for the field 
and KSDE to see which factors were the most influential over effectiveness 
rates, and which ones were inconsequential. In a way, the advance is in 
the collaboration with the field in identifying and testing their theories. 
They need to see the results so at least initially, we need to include the 
variables that were insignificant. 


Over time, we expect that the significant factors like cumulative poverty 
will stay in the model, but in the future, insignificant predictors may be 
dropped, and new variables, like measures of Individual Plans of Study, 
may be added, or tested for their influence over effectiveness rates. 


Ifa high school’s students are already ‘beating the odds,’ that is, already 
performing at the highest level possible, how will you know there isn’t any 
room for them to improve any more? Will the residuals still show some 
measure of performance ? 


KSDE answer: 


This is a question for all value-added models. They identify, quantify, and 
set aside the influence of some factors, like cumulative poverty, and 
assume that others, by subtraction, are in the residuals. Until we have 
measures that matter most in determining postsecondary attendance, the 
use of residuals will be subject to the suspicion that we might just be 
measuring natural variation, or mostly natural variation, in the residuals. 


Does the postsecondary National Student Clearinghouse data accurately 
describe postsecondary enrollments ? 


KSDE answer: 


Overall, the NSC data covers about 93 percent of students who actually 
continue in some postsecondary institution. There are some regional 
vulnerabilities where a high school might be located near one of the 
colleges or technical schools that doesn’t yet report its enrollments to the 
NSC but we are working to get all of these schools to begin reporting their 
enrollments to NSC. 


The bigger problem isn’t with the NSC data—we expect it to improve—but 
with the 30 percent or more of students leaving our high schools who are 


entirely unaccounted for. Ideally, we need post-high school income data. 
Anecdotally, almost everyone knows someone who was an academically 
poor high school student, but later economically successful. Most also 
know someone who was a good student, but met with economic problems 
after high school. Having income data would be a very big help. 


Are there incentives for manipulation of the data? The TAC noted that by 
including some factors over which schools and districts have some 
control, for example, suspension and expulsion rates, and chronic 
absenteeism, we may be creating incentives to manipulate the data. In a 
sense, by accounting for these factors as independent predictors, KSDE is 
forgiving them. 


KSDE answer: 


Ultimately, postsecondary enrollment rates will not be moved upward by 
manipulating the reporting of suspension and expulsion rates or absences. 
At most, manipulation of this reporting data can shift the benchmark, the 
predicted line, but to do so, either large numbers of schools would have to 
manipulate their data or small numbers would have to report extreme 
values, in which case the values become suspicious as outliers. 


The bigger danger is with certificates. Industry-recognized skill 
certificates can increase a school or district’s effectiveness rates. If a 
school or district awards or reports certificates that are not of real 
employment value for the students then the effectiveness rates can be 
inflated and the students harmed when they go into the job market. This is 
another reason that it is desirable to get income data, preferably federal 
Internal Revenue Service data, to validate the employment of students and 
the economic value of their preparation. 


You are trying to identify the influence high schools have over 
postsecondary enrollments. Why not control for academic performance in 
middle school? 


KSDE answer: 


Good idea. We think. We will add the cohorts’ mean combined math and 
reading eighth-grade state assessment scores into the model as an 
independent variable. This will control for previous academic 
performance, but will not account for the quality of middle schools’ career 
planning, career counseling, and hands-on curriculum and experiences. 
We expect these will still influence high-schools’ postsecondary 
enrollment rates. 


We aren’t sure we want the high schools to think of the results as 
reflective of their individual schools’ performances and not the result of a 
long effort, across the students’ school lives, by the elementary and middle 
schools that went before, and yes, the high schools, to prepare the students 
for success in the larger world. The Kansas Board’s vision is really to 
cultivate the whole student across their K-12 experience and into a 
successful transition into employment or postsecondary preparation. 


A few additional questions are included in sections where they are pertinent. 


Background & Policy Goals 


Why is KSDE defining K-12 performance measures for districts and schools that run two 
years after high school graduation? 


The intention is to bind district and school performance goals to the long-term interests of 
students and employers. By incorporating developmental measures across each child’s 
life, from Kindergarten readiness, through social-emotional skills, individual plans of 
study in middle and high school, and finally, to postsecondary and career goals, the State 
Board expects districts and schools to cultivate the life-success of each child across the 
students’ school life. To fully prepare youth for adulthood, the State Board is signaling 
the need for greater coordination between families, educators at all levels, and employers. 


This re-orientation is influenced by at least three sources: 


1. Labor-market studies show a mismatch and lack of coordination between the 
skills employers need and the skills new entrants to the labor market have. Two 
influential national labor market studies defined the size of this labor market / 
education gap for all the states.* The studies projected that, in order to supply the 
skills demanded by employers in 2020, the Kansas education system needs 71 
percent of young people entering the labor market to have an associate degrees or 
higher. Accordingly, the Kansas State Board of Education set a long-term goal of 
effectiveness rates between 70 to 75 percent. KSDE’s new postsecondary 
measure, the effectiveness rate, measures the gap between this goal and reality to 
be about 25 points wide. 


4 Anthony P. Carnevale, Nicole Smith, and Jeff Strohl at the Georgetown Public Policy Institute’s Center on 
Education and the Workforce published the two studies in 2013. The first paper, Recovery: Projections of 
Jobs and Education Requirements Through 2020, made detailed national job projections. The second 
paper has the same name but is called the State Report. It made state-by-state job projections. Of its 
new entrants to the labor market, the Kansas economy is projected to need 11 percent master’s degrees, 
25 percent bachelor’s degrees, and 35 percent associate degrees. International labor market studies 
confirm that in advanced as well as developing economies there is a global oversupply of unskilled labor 
which is disproportionately depressing youth’s economic prospects. See Dobb, R. et al, (2012), The world 
at work: Jobs, pay, and skills for 3.5 billion people, McKinsey Global Institute. Also see Dobbs, et al, (2016), 
Poorer than their parents? Flat or falling incomes in advanced economies, McKinsey Global Institute. 


2. A Kansas study that analyzed the responses of 287 focus groups in communities 
and business groups across the state, found that employers, educators, and 
participating members of the public emphasized that youth need social-emotional, 
character, personality, and other employability skills. Respondents cited the need 
for these soft skills much more frequently than traditional academic skills. 
Participants also strongly suggested greater collaboration, planning, and 
coordination between the K-12, higher education and business sectors. 


This popular assertion of the importance of social-emotional skills has been re- 
enforced by an increasingly strong current of economic and social research 
demonstrating that social-emotional, character, and personality skills are strong 
determinants of economic and life success.> 


3. Areaction to the design flaws of No Child Left Behind (NCLB). NCLB set rigid 
proficiency standards, untempered by what was demonstrably achievable. With 
high frequency, districts and schools serving economically disadvantaged families 
were identified as failing, even when they were performing much better than 
comparable organizations. At the same time, many districts and schools in 
advantaged communities exceeded the NCLB proficiency standard, but had little 
incentive to enhance the skills of their advantaged students beyond basic 
proficiency. 


How well do the new effectiveness measures address these goals and influences? 
The predicted effectiveness rates are based on the districts’ actual effectiveness rates. As 


a benchmark, it does not set goals that are unrealistically high. It draws the best-fitting 
line through the cloud of actual effectiveness rates. 


5 The return for employability skills, like conscientiousness, increases constantly across all skill levels. In 
addition to predicting higher wages, social-emotional skills are also associated with longer lives, lower 
divorce rates, and higher academic achievement at all levels. Another way of saying this is: no matter 
what your skill level, no matter what the job, social-emotional skills matter. 1Q matters little for low- 
skilled work, but matters a lot more for complex work. Another point made by this literature is that skills 
and knowledge are not separate, but intertwined, and hierarchically integrated through learning, practice, 
and experience. See p. 18 of Education for Life and Work: Developing Transferable Knowledge and Skills 
in the 21st Century, 2015, National Academy of Sciences. For a summary of studies showing the predictive 
relationship of non-academic skills to academic outcomes, see Poropat, A.E. (2009), “A meta-analysis of 
the five-factor model of personality and academic performance,” Psychological bulletin, 135, 322-338. 

For personality skills association with other life outcomes, see Roberts, B.W. et al (2007, December), “The 
power of personality: The comparative validity of personality traits, socioeconomic status, and cognitive 
ability for predicting important life outcomes,” Perspectives in Psychological Science 2 (4), 33-345. See 
also Heckman and Kautz (2013), “Fostering and measuring skills: Interventions that improve character and 
cognition,” NBER working paper, No. 191656. Using a national sample of employers from 1996, the 
authors found that 69 percent of the employers had rejected applicants because they lacked soft skills — 
like not missing work, showing up on time, or having a strong work ethic (conscientiousness). Employers 
rejected less than half as many applicants due to insufficient reading and writing skills. 


By quantifying and accounting for these influential factors that are beyond districts’ 
control, like cumulative poverty, the model reduces the influences that are confounded 
with the outcome variable, district effectiveness rates. 


The model also permits us to examine the relative performance of districts serving 
economically advantaged communities. It can answer the question, after accounting for 
District A’s low poverty and other relative advantages, is District A performing well 
relative to its peers? 


Precedents & Initial Requirements 


A Model Predicting Academic Performance 


The independent variables used in the effectiveness predictive model were first 
developed under the direction of a Kansas Learning Network (KLN) workgroup. In 2016, 
prompted by the obligations of the federal Every Student Succeeds Act (ESSA), the 
workgroup developed a method for identifying the neediest Title 1 schools.° 


Practitioners from the field identified nine factors that they believed would accurately 
predict low-academic performance in schools: 


chronic absence, 

student mobility, 

cumulative poverty, 

higher concentrations of Students with Disabilities, 

higher concentrations of migrant students, 

higher proportions of English Learners, 

the rate of suspensions and expulsions, 

the demographic distance in gender and ethnicity between teachers and students, 
and 

9. the percentage of new teachers. 


Oo Oe a 


The dependent variable was academic performance as measured by an index constructed 
from State assessments. Four of the nine, cumulative poverty, percentage of English 
Learners, the rate of suspensions and expulsions, and chronic absences, were significant 
predictors of lower school-level performance. They explained 58 percent of the variance 
in schools’ academic performance. 


Cumulative poverty was an especially strong detractor of school performance. When it 
was removed from the independent variables, the proportion of Students with Disabilities 
and student mobility changed from non-significant to significant predictors. 


6 The details of the CSI model are available online on the Research and Evaluation page at the KSDE site. 
See Moss, T. (2016), “A method for identifying Comprehensive Support and Improvement Schools and 
Holes in the Every Student Succeeds Act.” 


The evidence also suggested that at least two variables should be removed. The 
demographic distance between staff and students proved to be conceptually confused and 
without data to appropriately measure it. Migrant students were sparsely represented 
across districts and insignificant in their influence on schools’ academic performance. 
These two variables were removed from subsequent models. 


The KLN model served two purposes. It generated a formula that identified 
Comprehensive Support and Improvement (CSI) schools. The identified schools were 
urban schools with high levels of cumulative poverty and low levels of academic 
achievement. It also generated schools’ predicted academic achievement after controlling 
for the independent variables in the model. Similar to the procedures KSDE uses to 
identify Challenge grant schools that are performing well despite higher levels of student 
poverty, the model compared schools’ actual academic performance to their predicted 
academic performance. 


The First Model Predicting Effectiveness Rates 


In accord with the Board and KSDE’s new goals, leadership asked for a model that 
applied the risk factors developed with the KLN workgroup to a new dependent variable, 
the district effectiveness rates. Using the measures of the proportions of Students with 
Disabilities, chronic absence, English Learners, mobile students, cumulative poverty, 
suspensions and expulsions, and percentages of new teachers, this model explained 42 
percent of districts’ variance in effectiveness rates—a substantial amount but much less 
than the 58 percent of schools’ variance in academic performance explained by a very 
similar model. 


Only three variables were significant predictors: 


1. the proportion of Students with Disabilities, 
the proportion of mobile students, and 

3. cumulative poverty, which again, was strongly significant, somewhere beyond the 
0.001 level. 


The proportion of Students with Disabilities was not a significant predictor when the 
dependent variable was school academic performance as measured by state assessments. 


This first model told us that, on average, for every percentage point increase in Students 
with Disabilities, we could expect a district’s effectiveness rate to go down 0.265 points. 
In the average district, 16 percent of the students are Students with Disabilities, so our 
model predicts that the average district loses about four effectiveness points through 
influences associated with SwDs (16 * 0.265 = 4.3). In comparison, according to this 
first model, the average district loses about 17 effectiveness points due to cumulative 
poverty (43 * 0.404 = 17.4). 


When I removed cumulative poverty from this first model, the adjusted r-squared fell to 
0.25. Students with Disabilities then became the strongest predictor, followed by the 
proportion of English Learners and then mobile students. The suspensions and expulsions 
measure also became strongly significant, even though it had not been so when 
cumulative poverty was included. 


The Second Model 


After studying the results of the first model, KSDE leadership suspected that districts 
with virtual school programs had lower effectiveness rates, and that district size might 
also be a factor. A research review reported a correlation between the total number of 
schools attended and lower post-secondary enrollment. All three of these variables could 
be created from existing data. 


To test these suggestions and improve the explanatory power of the effectiveness model, 
I prepared a new set of independent predictors. I dropped the independent variables that 
were not significant in the first model. For the second model, I included the newly 

suggested variables and those that were significant in the first set, that is, the proportion 
of Students with Disabilities, the proportion of mobile students, and cumulative poverty. 


I also improved the accuracy of the some of the independent variables. I changed them 
from being based on whole district populations, to descriptors constructed from the five 
cohorts used to calculate the outcome variable, the districts’ effectiveness rates. 


For example, I first identified all students who had been classified as having a disability 
at any time in the school years from 2007 through 2015. The number of school years 
each student was so classified were added up across all available years of data from 2007 
through 2015. I also kept a cumulative total of the number of years each student had 
attended Kansas schools. 


Next, by year and grade, I selected the approximate cohorts in the postsecondary NSC 
data. All ninth graders from 2008 through 2012 were selected, all tenth graders from 
2009 through 2013, all eleventh graders from 2010 through 2014, and all seniors from 
2011 through 2015. I then aggregated, by district, the count of the total years these 
students were identified as having a disability. This quantity was the numerator. The 
denominator was the total number of years the selected students have attended Kansas 
schools. I used a similar process to calculate the virtual student rate, the cumulative 
poverty rate, the cumulative mobility rate, and the rate that students changed schools 
across their school careers. 


There is a subtle difference between the mobility rate and the rate that students changed 
schools. The mobility rate counts the number of times a student changes schools during 
the school year. The school transition rate counts the total number of times each student 
changed schools across their student lives. It includes the count of school transitions 
within the school year as well as those between the beginning and end of each school 


year. The two variables overlap, the total number of school transitions contains the total 
number of school changes within the school year, the mobility count. 


I used the count of the five cohorts of students aggregated to each district as a measure of 
district size. 


The second model was a substantial improvement from the first—it explained 53 percent 
of districts’ variance in effectiveness rates—an 11point improvement over the first 
model. Because of the high correlation between the mobility rate and the all-school 
transitions rate (see Table 1 below), I removed the latter from the model to avoid 
collinearity in the regression model.’ When I removed the mobility counts from the total 
school transitions counts, school transitions were not significant predictors of 
effectiveness rates. 


Table 1 


Bivariate Correlations Between 8 Measures 


1 2 3 4 5 6 7 8 
1. effectiveness rates 1 
2. district size 0.005 1 
3. SwD rate -.212"""  -0.084 1 
4. English Learner rate -.213°" -.12" -15° 1 
5. mobile rate -.427°° 0.07 37 0) 1 
6. all school transitions rate -.329"" 209°" 197" 0.03 .707° 1 
7. cumulative poverty rate -.667° 0.013.261.3905 37.297" 1 
8. virtual student rate -.272° (0.07 -0.1 -0.003 .322°° =.385° 0.04 1 


*** Correlation is significant at the 0.001 level (2-tailed). 
** Correlation is significant at the 0.01 level (2-tailed). 
* Correlation is significant at the 0.05 level (2-tailed). 


Which variables were significant predictors? We see in Table 2 below that only three 
were significant predictors: the student mobility rate, the cumulative poverty rate, and the 
virtual student rate. 


7 When both the mobility and the school change rate were included as independent variables in a 
regression procedure, the collinearity diagnostics returned some eigenvalues close to zero and a couple of 
conditional index values close or greater than 15. Both of these conditions indicate possible problems 
with collinearity; the mobility rate and school transitions rate were so similar that one could be predicted 
from the other. 


Table 2 


Coefficients from the Linear Regression Model Predicting District Effectiveness Rates 


Unstandardized Standardized 95.0% Confidence 
Coefficients Coefficients Interval for B 

Lower Upper 

B Std. Error Beta t Sig. Bound Bound 

(constant) 65.057 1.634 39.814 0.000 61.84 68.273 

district size ) ) 0.063 1.534 0.126 ) 0.001 
SwDrate -0.038 0.12 -0.014 -0.314 0.754 -0.273 0.198 

English Learnerrate 0.025 0.06 0.02 0.425 0.671 -0.093 0.143 
mobile rate -0.601 0.155 -0.176 -3.879 0.000 -0.906 -0.296 
cumulative poverty rate -0.5 0.043 -0.611 -11.699 0.000 -0.584 -0.416 
virtual studentrate -1.23 0.226 -0.229 -5.446 0.000 -1.675 -0.786 


The second model confirmed Commissioner Watson’s impression that districts with 
virtual students were more likely to have lower effectiveness rates. It also demonstrated 
that constructing the independent variables from the student freshman cohorts being 
measured in the dependent variable, the effectiveness rate, improved the accuracy of the 
model. 


After reviewing the results from the second model, KSDE leadership specified a third 
model. The model should include most of the independent variables used in the first 
model, variables that could be regarded as largely beyond the control of the districts: 


cumulative poverty; 

proportion of student years in which students were classified as having a 
disability; 

proportion of time students were classified as English Learners; 
proportion of student years with chronic absenteeism; 

proportion of mobile years; 

expulsion and suspension rates; and 

teacher turnover (percentage of new teachers). 


le 
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The model should exclude the proportion of virtual students and the new variables added 
to the second model. It should also produce a benchmark that would allow districts to 
judge their relative performance in effectiveness rates, one in which districts could have a 
high level of confidence, a 95-percent level of confidence. 


By including variables over which districts have limited influence, like chronic 
absenteeism, the percentage of new teachers, and the expulsion and suspension rates, 
does the model discourage districts from addressing these problems? 


Tentative Performance Categories 


KSDE staff have discussed the performance levels based on the standardized residuals 
but no decisions about them have been made. Below is one example, but other 
performance bands have also been discussed. 


e Far above average = = 1.5 standard deviations (sd) above the mean or predicted 
rate 

Above average = > | sd but < 1.5 sd above the predicted rate 

Typical or average = < | sd above and > -1 sd below the predicted rate 

Below average = < -1 sd below and > -1.5 sd below the predicted rate 

Far below average = < -1.5 sd below the predicted rate 


KSDE currently reports districts’ effectiveness rates based on the most recent five high 
school freshman cohorts available from NSC data. It reports the 95 percent confidence 
interval around each district’s predicted rate. It does not report the relative performance 
described by the categories above, but does explain how districts can calculate their own 
relative performance. 


Data Preparation for the Third, Current Model 


Student Longitudinal Records 


The student level data combines the audited enrollment records (ENR) from 2007 
through 2016 and the KIDS end-of-year (EOY) records. Why? 


The KLN workgroup had identified student mobility and chronic absenteeism as risk 
factors that were likely causes of poor academic performance. Mobile students and 
chronically absent students are relatively few and are more likely to have irregular school 
records. In order to accurately calculate rates of chronic absenteeism and mobility, we 
need highly accurate student records. 


The EOY records should contain all the students who attended school at any time during 
the year. The ENR records are audited and should contain all students who were 
attending the school on September 20". In comparing the two records, even after 
duplicate records were removed, one finds that about 1.76 percent of the ENR records are 
not included in the EOY records. To include those records, the ENR and EOY records 
were merged by student ID, year, and the accountability school. 


Before merging the EOY and ENR records, consistent categorical definitions were 
applied for students with disabilities, ethnic classifications, free or reduced or paid lunch 
status, EL classifications, including monitored, and virtual categories. Each record was 
labeled for its source, ENR or EOY. 


In order to merge the student records, duplicates had to be removed. When possible, 
responsible buildings’ IDs were assigned to student records without one. 


Separately, in the EOY and ENR records, student records without a responsible school, 
including those that were identified as being included in the accountability reporting for 
other states (Accountability School Identifier = ‘0001”), if they had an attending building 
in the record, were assigned a responsible school by making the attending building the 
responsible school. If no attending school was listed and the responsible school was listed 
as ‘0001’ or ‘0002’ (homeschooled), the record was removed. If the responsible school 
was missing and the attending building was missing, but the funding school was listed, 
the funding school was made the accountability building. 


The records were sorted by student ID, year, accountability school, grade, and the date 
the records were uploaded to KSDE. Duplicates were identified based on student ID, 
year, and accountability building. Then duplicate records were removed, but the most 
recently uploaded record was kept. This removed 686 records from the EOY files and 
1,969 from the ENR records. Since these records spanned 2007 through 2016, the 
removed records were tiny percentages of each record set. Nevertheless, adding the ENR 
records to the EOY records, after the duplicates were removed, added about 1.8 percent 
to the total merged records. 

About 4,500 records that were missing an accountability building but had a resident 
district were still in the file. These records were included in district aggregations to create 
the independent variables below. 


If grade was missing from the EOY record but available in the ENR record, the record 
was assigned the grade from ENR so that all records had a grade level. Based on the EOY 
grade, or the ENR grade if the EOY grade was missing, the records that were not graded 
(NG) and the pre-Kindergarten grades were removed. After sorting the records by student 
ID, year, load date, and accountability building, individual record numbers were assigned 
so that each student’s longitudinal records had an ordered sequence number. The 
maximum number of records for an individual student was 32. 


To calculate district aggregations, only public and private accredited districts were 
included (org_type = 2 or 5). Only districts reporting 12th graders were included. 


Independent Variables 


Cumulative Poverty 


Why include cumulative poverty? For a long time federal and state governments have 
measured relative student social-economic status (SES) using students’ eligibility for free 
or reduced lunch. Typically, they are used as cross-sectional, snapshot measures at 
enrollment or the end of the school year. At the school or district level, the count of 
students qualifying for either free or reduced lunch have the same value in the numerator. 
Schools that have a relatively high proportion of reduced lunch students in temporary 


poverty may have a poverty level comparable to those of schools with students who have 
lived in poverty and poor neighborhoods since birth. 


A better measure would capture the influence of the developmental stressors associated 
with poverty, especially in early childhood when those stressors are most influential.® 
These better measures of developmental stressors are not currently available. The closest 
available substitute is a measure of cumulative poverty, the total number of years a 
school or district’s students were in poverty divided by the total number of school years 
the students have attended state schools. 


With the merged files, I compared the ENR and EOY student classifications for each 
school attended and each year attended. When the free-lunch qualifications were missing 
from the EOY, but present in the ENR, I assumed the ENR classifications were correct 
and used them. When records from the two sources were not in agreement for a particular 
year, I chose the higher value based on this scale: 


free lunch = 1, reduced lunch = 0.5, paid lunch = 0. 


The above were also the values I assigned for each school year the student attended 
Kansas schools and qualified for free lunch (1) or reduced lunch (0.5). By starting from 
the first year each student attended Kansas schools, and adding the maximum poverty 
levels from each year to the one that followed it, I carried the sum total forward as a 
cumulative count of total time spent in poverty. I also created a cumulative count of the 
number of years each student attended Kansas schools. 


Using the cumulative values of student years spend in poverty as the numerator, and the 
cumulative count of student years spend in Kansas schools as the denominator, I 
aggregated the cumulative poverty levels for each district for each year. Because the 
cumulative poverty is attached to each individual record, when a student changes district, 
this quantity follows the student and is automatically carried forward to the district of 
present attendance. 


At the district level, cumulative poverty counts and years attended are systematically 
truncated. Most students who were kindergarteners in 2007, the first year of student 
longitudinal data that I’m including here, will be tenth graders in 2017. The total number 
of years they could have attended Kansas schools for this calculation is eleven. Though 
students have not attended Kansas schools their whole student careers, the poverty and 


8 Noble, K.G., Houston, S.M., Kan, E., & Sowell, E.R., (2012). Neural correlates of socioeconomic status in 
the developing human brain. Developmental Science, 15, 4, 516-527. See also Brooks-Gunn, J. & 
Markman, L. (2005). The contribution of parenting to ethnic and racial gaps in school readiness. The 
Future of Children, 15, 1, Spring, 139-168. Also Luby, J., Belden, A., Botteron, K., Marrus, N., Harms, M.P., 
Babb, C., Nishino, T., & Barch, D., (2013). The effects of poverty on childhood brain development. JAMA 
Pediatrics, 169 (10), 938-946. Dickerson, A., & Popli, G.K. (2016), Persistent poverty and children’s 


cognitive development: Evidence from the UK Millennium Cohort Study. Journal of the Royal Statistical 
Society A, 179, part 2, pp. 535-558. 


year counts are truncated in the same way for all districts. ’m assuming that this 
truncation doesn’t bias the district aggregates. 


Cumulative Years Students Were Classified as with Disabilities 


Should the proportion of students with disabilities be calculated like cumulative poverty 
is calculated, as a proportion of student’s time spent classified as having a disability? 
Intuitively, using longitudinal data to describe time spent in a particular status seems a 
more accurate model than a single cross-sectional snapshot of students’ status. In Table 1 
we saw that cumulative poverty has a strong correlation with disabilities. In the model 
predicting academic achievement, disability was not significant when cumulative poverty 
was included in the model, but became significant when cumulative poverty was 
removed. In the model predicting effectiveness rates, the disability measure became 
stronger when the cumulative poverty measure was removed. Cumulative poverty has a 
strong relationship with our disability measure. More time spent classified as having a 
disability may be indicative of the severity or persistence of impairment, and of its 
influence on a student’s development. 


If a student was identified as having a disability in either the EOY or ENR fields, for any 
of the schools attended in a given year, that student was counted as having a disability for 
that year (spednum = 1). The total number of years each student was identified as having 
a disability was added to each additional year attended and carried forward cumulatively. 


Similar to the cumulative poverty calculation, upon aggregation at the district level, the 
total count of student years spent classified as having a disability was classified as the 
numerator, and the total count of student years in Kansas schools was classified as the 
denominator. 


English Learner Rate 


Across all available student records, from 2007 through 2016, I identified records in 
which the student was identified as either currently receiving English language services 
or was a former EL student being monitored. (An EL student enters monitored status for 
the two years after the student has scored at the fluent level, level 4 or above, for two 
years in all domains and in the total score on the English Language Proficiency 
Assessment. These requirements were being reconsidered as this document was being 
written.) I aggregated the count of years in which the student was identified as either 
receiving EL services or being monitored. Like cumulative poverty and disability, this 
variable was calculated as a proportion of students’ time classified as an English Learner. 


Chronic Absence 


The way chronic absence was defined by the Kansas Learning Network workgroup 
differs from the current KSDE definitions. 


There are two federal definitions. The Civil Rights Data Collection (CRDC) defines 
chronically absent students as those who are: 


e “absent for 15 or more school days during the school year; 

e not physically on school grounds and not participating in instruction or 
instruction-related activities at an approved off-grounds location for the school 
day; and 

e absent for any reason (e.g., illness, suspension, the need to care for a family 
member).” 


The CRDC defines student absence in almost exactly the way KSDE’s Strategic 
Management Plan defines a student as absent: 


e “if he or she is not physically on school grounds and is not participating in 
instruction or instruction-related activities at an approved off grounds location for 
the school day.” 

e Absence includes “students who are absent for any reason (e.g., illness, 
suspension, the need to care for a family member), regardless of whether absences 
are excused or unexcused.” 

e Students who “miss 50 percent or more of a school day should be counted as 
absent.” 


KSDE, in its Strategic Plan Management Glossary, accepts the definition the federal 
Department of Education makes for its EDFacts data collection. The EDFacts data 
collection defines chronic absenteeism as the “unduplicated number of students absent 
10% or more school days during the school year.” Included are K-12 students “who were 
enrolled in the school for at least 10 school days at any time during the school year, and 
who missed 10% of the school days in which they were enrolled in the school.” 


The definitions include these rules: 


e “Students should be counted once at each school he/she attends.” 

e “Students should be counted in the chronic absenteeism data once they have been 
enrolled in a school for a minimum of 10 school days.” 

e “States should include state institutions (juvenile justice school and department of 
health services schools)...” 


At the time this model was being constructed, it wasn’t clear if the chronic absenteeism 
definition should have been changed to the KSDE /EDFacts definition above, or kept the 
same. No rule-making workgroups had met. The chronic absenteeism variable currently 
in the model is the KLN version. 


What is different about the chronic absence definition currently applied in the model? 
The KLN workgroup defined chronically absent students as those missing 10 days or 
more across all schools attended in a school year. Absences are calculated from the 
students’ point of view: they are cumulative so that a student who attends more than one 
school in the year carries forward the absences from previous schools to current schools. 


Here are the specific steps in the calculation as used in the predictive model described in 
this document: 


I first calculated the number of days each student lost in each school attended by 
subtracting the days attended from the days in membership. For those students who 
attended more than one school in a year, the lost days were carried forward to each 

school attended within the year. If the number of lost days equaled or exceeded 10 days 
across the year, then the record was identified as one in which the student was chronically 
absent in that year. Like cumulative poverty and the proportion of time classified as 
having a disability, I used an accumulated proportion of years with chronic absence in the 
current model. 


If the definition of chronic absenteeism is changed from the KLN definition to the KSDE 
/ EDFacts definition, and the years of chronic absence are cumulative, the performance of 
the variable as a predictor will probably be minimal. A small number of mobile students 
who are chronically absent across two or more schools will not be identified as 
chronically absent. If the rulemaking group changes the measure from a cumulative one, 
to a cross-sectional one, the impact on the predictive model might be greater, but this is a 
comparison that could be tested if a rulemaking group or leadership wants a comparison 
made. 


Mobile Students 


Student records indicating that a student had changed accountability schools within the 
academic year were flagged with a one. The number of times each student had changed 
schools within an academic year was aggregated at the student level based on each 
students’ ID. As with the variables above, the total number of years each student had 
attended Kansas schools was also aggregated, carried forward, and used to calculate a 
proportion of mobility years when the data were aggregated at the district level. 


Cohort Selection 


Once each student record had attached to it the cumulative count of years each student 
had: 


e qualified for free or reduced lunch, 
e been identified as having a disability, 
e received English Language services or been in EL monitored status, 


e missed more than 10 days within a year 
e been a virtual student or 
e changed schools within the school year 


I selected an approximation of the five freshmen cohorts and their peers. I selected: 


all records of 9"" graders in 2008 through 2012; 

all records of 10" graders in 2009 through 2013; 
all records of 11" graders in 2010 through 2014; 
all records of 12 graders in 2011 through 2015. 


Then I placed all the information needed from individual records in each individual 
students’ record. I selected the last available student record for each student so that there 
was only one record per student in the file. 


Next, I aggregated the records by district. I then used the aggregated counts of five risk 
factors as numerators over the total counts of each district’s student years. I multiplied 

the resulting ratios by 100, converting them into percentages. For the five cohorts, this 

produced these variables: 


cumpovrate: the percentage of students’ years spent in poverty. 

spedrate: the percentage of students’ years spent in disability status. 

esolrate: the percentage of students’ years spent in English Learner status. 

chronic2: the percentage of students’ years in which the student missed 10 or more days. 

mobile_rate: the cumulative count of school changes during the school year over the 
total number of years students have attended Kansas schools. 


There is another way that the accuracy of the independent variables might be improved. 
After approximating the five freshman cohorts, I should have aggregated the individual 
records to get all the building IDs of the schools the five freshmen cohorts attended from 
2008 through 2015. Let’s call this dataset X and set it aside. When it comes time to 
calculate the percentage of district teachers who are new to their schools, we should first 
filter the buildings by dataset X. Then when we aggregate the counts of teachers who are 
new to their schools, and the number of teachers in those schools, we are not aggregating 
across all schools in each district, but only by those schools attended by the five freshman 
cohorts. 


Expulsion and Suspension Rates 


For the five-year period the five cohorts were attending schools, KSDE does not have 
individual records for suspensions and expulsions.’ For 2012 through 2017, it has 
building-level yearly counts of in-school and out-of-school suspensions, as well as 
expulsions. There are other, non-suspension and non-expulsion discipline events, and 
truancies, but the KLN workgroup chose not to include them in this calculation. 


I removed three duplicates at the building and year levels. Unlike the other independent 
variables above, district suspension and expulsion rates are a general district description 
across all students. We cannot calculate suspension and expulsion rates from the five 
freshman cohorts. How much noise this makes, and whether this noise obscures the 
signal from this measure cannot be quantified until KSDE acquires a measure of 
discipline events that is tied to individual student records. 


To match the time frame of the five freshman cohorts as closely as possible, I selected the 
2012 through 2015 years and aggregated the counts of these discipline events at the 
district level. This was the numerator. The denominator was the count of students from 
the audited enrollment counts across the same years for each district. As with the other 
variables, I converted this proportion to a percentage. 


It is possible to apply the same filter, the dataset X, to the expulsion and suspension data. 


Teacher Turnover (Percentage of New Teachers) 


This is the one independent predictor that isn’t based on student-level data. There is some 
research identifying teacher turnover as a factor in lower student performance. Studies of 
international education systems have sometimes pointed to high teacher turnover in the 
United States as an important factor in comparatively lower student performance.'° 
Domestic studies also note the high costs of teacher turnover and use teacher mover and 
leaver rates as measures.!! 


° KSDE began collecting some suspension and expulsion data tied to individual student records in 2017. 
KSDE does not collect all suspension, expulsions, and discipline events, only those required for federal and 
state reporting. 

10 Auguste, B., Kihn, P. and Miller, M. (2010). Closing the talent gap: Attracting and retaining top-third 
graduates to careers in teaching. Washington, DC: McKinsey. 

http://www.mckinsey.com/App Media/Reports/SSO/closing the talent_gap september 2010.pdf. 


™ Kukla-Acevedo, S. (2009). Leavers, movers, and stayers: The role of workplace conditions in teacher 
mobility decisions. The Journal of Educational Research, 102, 6. Also see Ingersoll, R.M. (2001). Teacher 
turnover, teacher shortages, and the organization of schools. University of Washington: Center for the 
Study of Teaching and Policy. Retrieved from http://depts.washington.edu/ctpmail/PDFs/Turnover-Ing- 
01-2001.pdf. 


The KLN workgroup chose to measure a five-year average of the percentage of new 
teachers in a district. They felt the five-year average would better identify systemic 
teacher turnover and avoid identifying small schools with the misfortune of having a high 
proportion of retirements in a single year. 


From the Educators Assignment database, I acquired the staff ID, program year, district 
and building assignments, teacher codes, and teacher code descriptions. I selected only 
teachers and excluded administrators and school specialists (teacher_code = 1, 2, 3, 4, 5, 
or 9). [removed duplicate records (records with the same staff ID, year, and school ID). I 
assigned individual record numbers for each record tied to an individual staff ID. I 
marked the first individual teacher records as new to the building. I tagged each record 
that marked a teacher’s first year in a school (firstyearinbldg). Those sequential 
individual records in which staff IDs were the same, and the building numbers were the 
same, but the year had advanced by one, I marked as a teacher continuing within the 
same school (continuinginbldg). In order to understand the records and avoid 
mislabeling some, I also classified and tagged the records showing teachers who rotated 
between buildings within the year (rotatebldg). Finally, I identified those sequential 
records where the staff IDs were the same but the school ID had changed, and the record 
was not marked as either a case of continuing in the building nor as a building rotation. I 
labeled these records as identifying teachers who were new to that building (newtobldg). 


I also labeled records using the labels established by scholars, “movers,” and “leavers.” I 
identified six categories of movers, and two categories of leavers. 


If we want to reduce the noise in the measure of teachers who are new to their buildings, 
we will retrieve dataset X and use it to filter the buildings used in the teacher 
aggregations. Dataset X identified the schools the five cohorts were attending from 2008 
through 2015. 


I then aggregated these counts by district and year so that I had counts of the total number 
of teachers, and the total number of teachers in each of the categories I had created. 


With a more complete data set, we would be able to match the individual students in the 
five freshmen cohorts to all the teachers they have had through their school careers and 
identify the proportion of their teachers who were new. KSDE does not currently have 
the data to tie particular courses, teachers, and years to particular students. This would 
have given us a more accurate answer to the question, Does the proportion of new 
teachers influence a district’s effectiveness rates? 


In accord with the original specifications of the KLN workgroup, I aggregated the 
following district counts within the five-year time frames below: 


total_teachers the count of teachers in district 


newtobldg the count of teachers new to the school in which they teach 

rotators the count of teachers who rotate between schools within the same year 
leaverlstyr the count of teachers who leave teaching in Kansas after their first year 
leavertotal the count of teachers who leave teaching in Kansas 

movedwi the count of teachers who moved between buildings within the district 
moveddist the count of teachers who moved from one district to another 
movertotal total movers from one building to another 


So that questions about trends can be asked and answered, I made district aggregates 
within the five-year time frames below. I labeled each with the highest year included in 
the time frame: 


2003-2007 2007 
2004-2008 2008 
2005-2009 2009 
2006-2010 2010 
2007-2011 2011 
2008-2012 2012 
2009-2013 = 2013 
2010-2014 2014 
2011-2015 = 2015. 


I used only the 2011 through 2015 dataset to create the new teachers’ variable. As I’ve 
explained above, it may improve the accurate of this independent variable if the data 
were filtered to include only the buildings the five cohorts had actually attended. It may 
also help to expand the time frame for the aggregation from 2008 through 2015. 


Once the teacher dataset is filtered by the selected time frame, the next step is to create 
the district aggregations of the teacher variables. I calculated the following percentages: 


pctnewtobldg = (newtobldg / total_teachers) * 100 = percent of teaching staff new to the 
building 

pct_rotators = (rotators / total_teachers) * 100 = percent of teachers who are rotating 
between buildings 

pct_leavelstyr = (leavr1styr / total_teachers) * 100 = percent of teachers who have left 
after their first year 

pct_leavers = (leavertotal / total_teachers) * 100 = percent of teachers who have left 
teaching or KS 

pct_movedwi = (movedwi / total_teachers) * 100 = percent who have changed buildings 
within the district 

pct_moveddist = (moveddist / total_teachers) * 100 = percent who what changed 
districts within the State 

pct_movertotal = (movertotal / total_teachers) * 100 = percent who have changed 
schools 


This produced these eight independent variables at the district level: 


— 


the proportion of years a district’s students were virtual students; 

the cumulative proportion of years that students’ families were low income; 
the cumulative proportion of years that students were classified as having 
disabilities; 

the proportion of years students were classified as English Learners; 

the proportion of years that students were chronically absent; 

the proportion of years students had been mobile within the school year; 

a measure of district’s expulsion and suspension rates; and 

a measure of the proportion of new teachers. 
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Dependent Variable 


Effectiveness Rates 


KSDE’s new postsecondary measures are based on longitudinal student records. KSDE, 
as part of its calculation of high-school graduation rates, identifies ninth-grade student 
cohorts. In order to calculated graduation rates according to the required federal formula, 
KSDE follows the progress of these cohorts from grade to grade and through high school 
graduation. KSDE then sends the graduation records to the National Student 
Clearinghouse (NSC). NSC links the high school graduate records to college enrollment 
records. It then sells these records back to KSDE. KSDE uses the records to produce two 
postsecondary performance measures. 


1. The success rate uses the count of graduates from the four-year ninth-grade 
cohort, adjusted for transfers in and out, as its denominator. Its numerator is the 
count of high-school graduates who have been enrolled in postsecondary 
education for two years, or, who have already completed a postsecondary degree 
or credential. 


2. The effectiveness rate uses the adjusted ninth-grade cohorts, the same ones used 
in the high-school graduation rates, as its denominator. The numerator is the 
count of students from the same cohort who have been continuously enrolled in 
postsecondary education for two years, or have already completed a 
postsecondary degree or credential. 


Because effectiveness rates include students who eventually will drop out of high school 
in its denominator, or who graduate but do not go on to postsecondary, effectiveness rates 
are typically eight or more points lower than success rates. Effectiveness rates offer a 
more complete picture of how effective institutions are in preparing and guiding youth 
through high school graduation and into postsecondary training. 


The effectiveness rate is also sometimes described as the graduation rate multiplied by 
the success rate. But as Sheng Xuewen, a senior KSDE researcher points out, two of the 
terms cancel, so we are left with the effectiveness rate as described above. 


graduation rate * success rate effectiveness rate 


count of h.s. graduates count of h.s. graduates 
-count of graduates— m enrolled in postsecondary enrolled in postsecondary 
count of freshmen “count of graduates— count of freshmen 


The formula KSDE currently uses is below. To make it clear, I’ve broken it into two 
steps: 


graduation rate = (dgsr_student_grads / dgsr_student_totals) where 


dgsr_student_grads = student count of the adjusted freshman cohort who have 
graduated, and 


dgsr_student_totals = student count of the adjusted freshman cohort. 
success rate = ((ps_retained_count + ps_grad_count) / nsc_total_in_class) where 


ps_retained_count = count of students in the adjusted freshman cohort enrolled in 
a postsecondary institution two years after high-school graduation, 


ps_grad_count = count of students who have graduated from high school and a 
postsecondary program or have an industry-recognized skill certificate, and 


nsc_total_in_class!? = count of high-school graduates in NSC’s records; 
and then, success rate * graduation rate = effectiveness rate. 


As I’d done with the independent variables, I converted the effectiveness rates to 
percentages. 


KSDE treats the combined five freshman cohorts as though they were a single cohort. It 
aggregates student counts from all five years. It does not calculate each year separately 
and then average the five effectiveness rates. 


Why does KSDE aggregate the most recently available five high-school cohorts in its 
calculation of effectiveness rates? Kansas has many small, rural districts. In order to get 


12 Theoretically, KSDE could use its count of high school graduates here. This would simplify the 
calculation to: ((ps_retained_count + ps_grad_count) / dgsr_student_totals) = effectiveness rates. When 
subtracting the NSC high school graduates from the KSDE graduates, there are 183 districts that do not 
match. The differences range from -10 to 69. 


stable effectiveness rates, KSDE aggregates the graduates from the most recent five years 
available. As of the writing of this document, the aggregates include the high school 
freshman cohorts of 2011, 2012, 2013, 2014, and 2015. 


Applying OLS Linear Regression 


In SPSS, I ordered a frequencies table and found that some records were missing values 
for the disability measure. So that these cases would be included in the regression 
procedure, I replaced the missing values with zeros. Because of the way I had constructed 
the independent variables, there were no independent variables with missing values. 


I ordered a correlations table and saw that the mobility measure had a high correlation 
(0.65) with the school change rate variable. The school change and the virtual variables 
were not part of the risk model leadership specified for generating the predicted 
effectiveness rates, so they were not included in this model. 


With the variables assembled as columns in a single SPSS file with each accredited 
district as a row, I used linear regression with the ENTER method. It enters the 
independent variables as a block, all at once. I set missing values to be deleted listwise, 
but, as I’ve noted, there were no missing values except for three tiny districts without 
effectiveness rates. I ordered a model summary table with the adjusted r-squared and the 
standard error of the estimate, regression coefficients, and a collinearity diagnostics table. 
I saved the predicted values, the residuals (the predicted — the actual effectiveness rates), 
and the 95 percent confidence interval around the mean predicted effectiveness rates. 


Output for Reporting & Technical Documentation 


Table 3 


N, Medians, Means, Standard Errors, Standard Deviations, and Ranges of the 9 Variables 


standard standard 

N median mean error deviation range 
1. effectiveness rates 286 45.1 45.81 0.585 9.89 68.2 
2. proportion of student years SwD 289 12.59 12.88 0.332 5.64 83.6 
3. proportion of student years English Learners 289 0.575 3.87 0.501 8.53 57.1 
4. cumulative rate of mobility 289 3.89 4.52 0.195 3.32 30.2 
5. proportion of student years in poverty 289 32.62 34.23 0.726 12.35 74.1 
6. proportion of student years as virtual 289 0.211 0.629 0.109 1.85 17.5 
7. expulsion & suspension rates 2012 - 2015 289 3.18 4.74 0.341 5.8 47.9 
8. percent of new teachers 289 12.66 13.91 0.374 6.35 59.3 
9. chronic absence rate based on years attended 289 29.86 30.42 0.482 8.19 60.9 


Table 3 above describes the variables in the regression model, plus the virtual variable, 


which is not in the model but is included in Tables 3 and 4 to quantify its influence on 


effectiveness rates. 


Table 4 below has the correlations between pairs of variables. The row numbers 
correspond with the column numbers—for example, column one shows the correlations 
between effectiveness rates and every other variable. Since a variable correlates 


completely with itself, the intersections where the column and row represent the same 


variable are ones. 


Note that there are several cells without any superscript asterisks. This means there is no 
significant correlation between the two intersecting variables. 


Table 4 
Bivariate Correlations Between the Prepared Measures 


1 2 3 4 5 6 7 8 9 
1. effectiveness rates 1 
2. proportion of student years SwD -.240** £ 
3. proportion of student years English Learners -.223** — -.146* 1 
4. cumulative rate of mobility -.418**  .396** -0.006 1 
5. proportion of student years in poverty -.659** = .220** ARS**. | {321** 1 
6. proportion of student years as virtual -.271** — -0.086 -0.013 agi 0.003 1 
7. expulsion & suspension rates 2012 - 2015 -.282** — -0.040 -0.083 0.047 .248** -0.001 i 
8. percent of new teachers -.130* .146* 0.062 .189** .138* .132* 0.057 1 
9. chronic absence rate based on years attended = -.387** 0.046 0.100 si7ar* .364** 0.114 0.114 152** 1 


** Correlation is significant at the 0.01 level (2-tailed). 
* Correlation is significant at the 0.05 level (2-tailed). 


All of the independent variables have a significant correlation with effectiveness rates, 
the dependent variable. I have included the proportion of student years that were spent as 
virtual students in this table even though it will not be an independent predictor in the 
regression model. We can see that it has a strong correlation with lower effectiveness 
rates. Cumulative poverty has the largest negative correlation with effectiveness rates and 
strong correlations with all of the independent variables except the virtual variable. 


Table 5 
Model Summary” 
Std. Error of the 
R R Square Adjusted R Square Estimate 
117° 0.514 0.502 6.98 


a. Predictors: (Constant), chronic absence rate based on total number of years 
attended, % of teaching staff new to the building, proportion of years students 
spent in SwD status, expulsion & suspension rate based on 2012 thru 2015 counts, 
cumulative rate of mobility for 5 yr cohorts, proportion of student years spent in EL 
status, proportion of years students in poverty status 


b. Dependent Variable: effectiverate2 


In Table 5, the model summary tells us that our independent variables are explaining 
half—0.5—of the variance between districts effectiveness rates. Explaining half is good 
for a beginning model, but also humbling in that there is an equal amount of the variance 
that the model doesn’t explain. 


The model also reports the standard error of the estimate as about seven (6.98). We can 
think of this as a measure of how districts’ actual effectiveness rates are distributed 
around the prediction line. We will use this distance to make the tentative performance 
categories described on page eleven. About seven point above or below the predicted line 
is within one standard deviation away from the predicted line. 


Below, table 6 gives us the results of our linear regression model. It tells us which 
variables are significant predictors of effectiveness rates. 


Table 6 
Coefficients from the Linear Regression Model 


Unstandardized Standardized 


Coefficients Coefficients 

B Std. Error Beta t Sig. 
(Constant) 71.336 2.261 31.554 0.000 
proportion of student years SwD -0.110 0.126 -0.041 -0.870 0.385 
proportion of student years English Learners -0.031 0.064 -0.025 -0.488 0.626 
cumulative rate of mobility -0.753 0.155 -0.220 -4.856 0.000 
proportion of student years in poverty -0.390 0.049 -0.478 -7.960 0.000 
expulsion & suspension rates 2012 - 2015 -0.197 0.078 -0.116 -2.536 0.012 
percent of new teachers -0.095 0.075 -0.054 -1.267 0.206 
chronic absence rate based on years attended -0.172 0.058 -0.136 -2.988 0.003 


a. Dependent Variable: effectiveness rates 


Table 6 gives us the coefficients or weights that optimally predict district effectiveness 
rates. There are many types of regression. The linear regression used here is 


straightforward: 


Y'=bo+ bi x1 + bo x2 + b3 x3 + bg x4 + bs X5 + Do X6 + 67 X7+ C 
Where Y' = the predicted or estimated effectiveness rate 
bo = the constant, y-intercept, or grand mean 


b; = the coefficient or weight for the first independent variable 


X1 = the proportion of student years spent classified as having a disability 
and so forth across the next six independent or predictor variables to e, the remaining 
error. 
How should we interpret these results? 


Looking in the column marked “Sig.” for significance, we see that, after the constant or 
y-intercept, four variables have significance levels at or smaller than 0.05: 


the proportion of student years spent in poverty 
cumulative rate of mobility 

the chronic absence rate, and 

the expulsion and suspension rates. 


PP Oe 


The other variables, the proportion of student years spent classified as having a disability, 
the proportion of student years classified as an English Learner, and the percentage of 
new teachers, are not significant predictors of effectiveness rates. 


How do we interpret the unstandardized regression coefficients? 


For the four significant predictors, we can say that for every unit change in the 
independent variable, the dependent variable changes by the coefficient’s units. For 
example, for every percentage point increase in cumulative mobility, we can expect the 
effectiveness rate to go down about 0.75 points. In Table 3, we see that in the average 
district, 4.5 percent of the student years are mobile years, so, on average, our model 
predicts that the average district loses a little more than three effectiveness points through 
the influences associated with student mobility (4.5 * 0.75 = 3.4). In comparison, the 
average district loses about 13 effectiveness points due to the influence of cumulative 
poverty (34.2 * 0.39 = 13.3), about 5 points due to chronic absenteeism (30.4 * 0.17 = 
5.2), and about a point due to expulsion and suspension (4.7 * 1.97 = 0.9). 


These results suggest that the largest gains in effectiveness rates may be in policies and 
programs that counter the effects of cumulative poverty, followed by absenteeism, 
mobility, and finally, expulsion and suspension. 


What happens if we remove the measure of cumulative poverty? 


The explained variance, the adjusted r-squared, drops to 0.39 and all our remaining 
measures, except for the percentage of new teaching staff, become strongly significant. 
Table 7 below shows the results. The evidence is again telling us how important 
cumulative poverty is, and how influential it is with our other measures. 


Table 7 
Coefficients from the Linear Regression Model with Cumulative Poverty Removed 


Unstandardized Standardized 
Coefficients Coefficients 
B Std. Error Beta t Sig. 
(Constant) 70.597 2.499 28.255 0.000 
proportion of student years SwD -0.548 0.126 -0.207 -4.358 0.000 
proportion of student years English Learners -0.303 0.060 -0.243 -5.082 0.000 
cumulative rate of mobility -1.140 0.163 -0.334 -7.003 0.000 
expulsion & suspension rates 2012 - 2015 -0.410 0.080 -0.241 -5.097 0.000 
percent of new teachers -0.087 0.083 -0.049 -1.052 0.294 
chronic absence rate based on years attended -0.282 0.062 -0.222 -4.564 0.000 


a. Dependent Variable: effectiveness rates 


What do the residuals and predicted effectiveness rates look like? 


Table 8 
N, Medians, Means, Standard Errors, Standard Deviations, and Ranges of the Predicted and Residual Effectiveness Rate 
standard standard 


N median mean error deviation range 
predicted effectiveness rates 289 46.02 45.55 0.438 7.45 45.39 
residual rates (actual - predicted = residual) 286 -0.107 (0) 0.406 6.87 40.87 


The residuals have a mean of zero, and standard deviations of about seven, as we would 
expect. 


